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About the National League of Cities Institute for Youth, Education and Families 

The Institute for Youth, Education and Families (YEF Institute) is a special entity within the National League of Cities 
(NLC). 

NLC is the oldest and largest national organization representing municipal government throughout the United States. Its 
mission is to strengthen and promote cities as centers of opportunity, leadership and governance. 

The YEF Institute helps municipal leaders take action on behalf of the children, youth and families in their communities. 
NLC launched the YEF Institute in January 2000 in recognition of the unique and influential roles that mayors, city 
councilmembers and other local leaders play in strengthening families and improving outcomes for children and youth. 

Through the YEF Institute, municipal officials and other community leaders have direct access to a broad array of strategies 
and tools, including: 

• Action kits that offer a menu of practical steps that officials can take to address key problems or challenges. 

• Technical assistance projects in selected communities. 

• Peer networks and learning communities focused on specific program areas. 

• The National Summit on Your City’s Families and other workshops, training sessions and cross-site meetings. 

• Targeted research and periodic surveys of local officials. 

• The YEF Institute’s website, audioconferences and e-mail listservs. 

To learn more about these tools and other aspects of the YEF Institute’s work, go to www.nlc.org/iyef. 


About Harvard Family Research Project 

Harvard Family Research Project (HFRP), based at the Harvard Graduate School of Education, researches, develops and 
evaluates strategies to promote the well-being of children, youth, families and their communities. HFRP works primarily 
within three areas that support children’s learning and development: early childhood education, out-of-school time 
programming and family and community support in education. 

Underpinning all of HFRP’s work is a commitment to evaluation for strategic decision making, learning and accountability. 
Building on the knowledge that schools alone cannot meet the learning needs of our children, HFRP also focuses national 
attention on complementary learning. Complementary learning is the idea that a systemic approach, which integrates 
school and nonschool supports, can better ensure that all children have the skills they need to succeed. 

To learn more about how HFRP can support your work with children and families, visit www.hfrp.org. 


About The Wallace Foundation 

The Wallace Foundation is an independent, national foundation dedicated to supporting and sharing effective ideas and 
practices that expand learning and enrichment opportunities for all people. Its three current objectives are: strengthening 
education leadership to improve student achievement; enhancing out-of-school learning opportunities; and building 
appreciation and demand for the arts. More information and research on these and other related topics can be found at 
www.wallacefoundation.org. 


About this Guide 

Lane Russell, associate for afterschool at NLC’s Institute for Youth, Education and Families, and Priscilla Little, former 
associate director at Harvard Family Research Project, and currently an independent research and evaluation consultant, 
served as the authors of this strategy guide. Audrey M. Hutchinson, the YEF Institute’s program director for education and 
afterschool initiatives, provided guidance and oversight to the development of this strategy guide and Clifford M. Johnson, 
the institute’s executive director, provided overall editorial direction. Michael Karpman provided additional editorial 
support. Alexander Clarke was responsible for the strategy guide’s design and layout. Preparation and distribution of this 
strategy guide were made possible by support from The Wallace Foundation. 
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Executive Summary 

Cities around the country are building systems that seek to make the most of public and private resources to provide 
widespread, high-quality, out-of-school time opportunities. Local officials are leading these efforts because partici- 
pation in OST programs not only benefits young people but also the cities in which they live, with the potential to 
support many city priorities — from reducing crime to creating a more skilled workforce. However, even with grow- 
ing public awareness of the importance of OST as a learning and developmental support, millions of children, espe- 
cially the neediest, are missing out on these opportunities, with an estimated 15.1 million children returning to an 
empty house at the end of the school day. 1 

The success of citywide systems in broadening access to high-quality OST programs for underserved children and 
youth often depends on the effective use of data. Reliable data enable city leaders to measure the scope and impact 
of local OST programs across their communities and to promote continuous improvement within these programs. 
For instance, geographic information system (GIS) data can help municipal officials determine whether programs 
are reaching youth in disadvantaged neighborhoods, while surveys, focus groups and attendance information can 
reveal whether programs fit the interests and needs of youth and their parents. Municipal officials can also partner 
with school districts and other entities, linking data across multiple sectors to assess the impact of OST programs on 
academic performance or other key indicators of well-being. 

Although strengthening local capacity to collect and analyze data can be challenging, cities reap multiple benefits 
from increased access to reliable information. One benefit is an enhanced ability to steer limited resources toward 
high-quality programs. Participant information can shape decisions about which programs are effective at attract- 
ing and retaining participants and which programs may need to be discontinued. Data-driven approaches are also 
advantageous in generating widespread support among residents for needed services. Communicating the impact 
and cost-effectiveness of OST programs — as well as the existence of gaps in services — helps build public will for 
supportive policies and investments. A number of cities find data critically important in their efforts to develop 
quality standards for local programs and professional development opportunities to help program staff meet those 
standards. Finally, mechanisms for shared accountability that rely on data help sustain the partnerships among cit- 
ies, school districts and nonprofit organizations that are necessary for creating seamless learning opportunities for 
young people throughout the community. 

This guide provides municipal leaders and their key partners with strategies for collecting and using information to 
strengthen citywide OST systems. The six strategies presented here are intended to help build the capacity of com- 
munities to get and use information across a citywide system: 

1. Collect attendance and participation information to support access, enrollment and quality improvements; 

2. Conduct market research to assess family, youth and community needs; 

3. Use geographic information system (GIS) technology to identify service gaps and improve access; 

4. Develop, adapt and implement a citywide program quality assessment tool; 

5. Conduct and support program and city-level evaluation efforts; and 

6. Develop common, system-wide outcome measures/indicators. 

The guide presents examples of how mayors, councilmembers and other municipal leaders across the country have 
implemented each strategy to improve the OST opportunities available in their cities and towns. Their efforts high- 
light the powerful role city leaders can play in supporting data collection at the program and city levels. Their work 


1 Afterschool Alliance (2009). America Alter 3PM. Washington, DC: The Afterschool Alliance. 
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has also strengthened the research base demonstrating the benefits of OST programs and has shown that informa- 
tion, if used well, can help municipal leaders be strong advocates and champions for sustained public investment in 
high-quality, out-of-school time programs. 


STRATEGY 1: COLLECT ATTENDANCE AND PARTICIPATION INFORMATION TO 
SUPPORT ACCESS, ENROLLMENT AND QUALITY IMPROVEMENTS. 


Perhaps the single most important piece of information cities can require all programs to track is attendance. With- 
out a good understanding of who attends programs and how frequently, an OST program is unable to make credible 
claims about its effectiveness and a citywide system cannot effectively monitor its programs. Therefore, collecting 
and understanding attendance data is important to both programs and cities alike. Attendance data allow municipal 
leaders and their partners to: 

• Gauge demand for services and better target access and enrollment efforts; 

• Support program planning and management by identifying popular and poorly attended programming; 

• Connect individual youth to other programs and services; 

• Monitor program quality; 

• Fulfill accountability requirements tied to average daily attendance; and 

• Develop a data-driven advocacy and sustainability strategy. 

When participation information is fed into a citywide tracking system, it can help identify service delivery gaps in 
specific geographic locations across the city. While programs may be using a variety of methods to measure atten- 
dance, municipal leaders can require all programs operating in their city to collect the same basic attendance infor- 
mation and provide that information to a citywide tracking system. 

Cities are increasingly using management information systems (MIS) to collect participant information as a cen- 
tral component of city-level system building work. Throughout a community, there are a number of organizations 


How Should Programs Measure Attendance? 

Most researchers agree that programs should collect the following information regarding attendance: site 
name; total number of students enrolled; total head count per day, week and year; student names; student ID 
number; age/grade in school; first/last enrollment date; and demographic information. 

How do programs measure attendance for reporting purposes? There are at least three different ways that 
programs report attendance data: 

1. Average daily attendance (ADA), the total number of children who come in the door 

2. Number of days each child attends 

3. Number and types of activities that each child attends 

Of these three types, ADA is the most common attendance measure for citywide systems to request, but 
is the least useful measure for programs that want to use their attendance data for program planning and 
improvement. 

Adapted from Fiester, Leila, with Policy Studies Associates, Inc. (2004). Afterschool Counts! A Guide to Issues and Strategies for 
Monitoring Attendance in Afterschool and Other Youth Programs. New York, NY: The Robert Wood Johnson Foundation. Retrieved 
from http://www.theafterschoolproject.org/uploads/After_School_Counts-vl.pdf. 


1 



NLC INSTITUTE FOR YOUTH, EDUCATION, AND FAMILIES STRATEGY GUIDE 



(e.g., schools, libraries and social service agencies) that use MIS to collect youth data — including attendance, grades, 
demographics and risk behavior statistics. Sharing this information across organizations can help build an effective 
citywide out-of-school time system, ensure that the necessary supports for youth are in place and demonstrate OST 
programs’ value. Municipal leaders can encourage collaboration with schools and community partners to combine 
participant information. While building and strengthening these partnerships for sharing data can be a complex 
task, a sustained effort can serve to strengthen programs. 

Some cities’ management information systems include a program locator that can identify underserved areas of the 
city as well as places where providers are potentially competing for participants. Other cities use their MIS for program 
improvements, monitoring attendance patterns as crude measures of program quality to help identify strong programs 
and programs in need of improvement. When local leaders collect attendance data, they often set thresholds of partici- 
pation for accountability purposes so that attendance rates are not skewed by a high number of low participators. For 
instance, programs may be asked to report data only on young people who participate in programs two or more days per 
week. Monthly reports enable system leaders to examine attendance patterns, investigate why some programs are better 
attended than others and consider program improvements. Some city officials opt to engage firms that can customize a 
pre-existing software package to their city’s needs. Others opt to design and build a data system that is unique to their city. 

Despite its recognized benefits, collecting attendance data can be challenging and municipal leaders need to keep 
some barriers in mind. For instance, OST program staff often do not have the training or capacity to use MIS com- 
puter technology effectively to measure and track meaningful attendance information. One way to address the lack 
of staff capacity is for municipal leaders to work with their partners to implement a professional development and 
training structure for end users on how to enter and use participant data. It is also critical to keep legal issues and 
privacy considerations in mind when sharing data across organizations. For instance, working in collaboration with 
libraries can pose significant privacy barriers because federal laws prohibit libraries from sharing participant data 
with outside entities. When discussing data sharing between systems, municipal leaders, school district officials and 
other community leaders can work together to ensure compliance with legal requirements that protect the privacy 
needs of young people while also collecting the data that will help improve the programs serving these children. 

Louisville, Kentucky 

Since 2001, Louisville Metro Government’s Office of Youth Development has partnered with Jefferson County Public 
Schools (JCPS), Metro United Way and other community-based organizations on the KidTrax partnership, which 
provides a data and tracking tool. The TraxSolutions (formerly KidTrax) system allows community-based organiza- 
tions and schools to share and access aggregate and individual data on the youth they serve. Students participating 
in afterschool programs scan their bar-coded TraxSolutions cards, which also serve as library cards and bus passes. 
The cards enable local partners to gather students’ individualized data via computer software. The city requires 
the community-based programs to which they provide funding to implement TraxSolutions and share data with 
JCPS. Additionally, the city provides outcome measurement training and assists community-based organizations in 
measuring program outcomes through TraxSolutions. TraxSolutions is also helping the city develop and measure 
community-wide outcomes for young people as it works toward developing a comprehensive vision for youth. 

In addition to monitoring program participation, TraxSolutions software (developed by nFocus) integrates JCPS data 
on test scores, truancy rates and suspensions. This system not only allows partners to assess the effects of programs 
on student achievement; it also helps educators and youth service providers work together to intervene on behalf of 
youth who are struggling in school. With the data collected through TraxSolutions, JCPS conducted research on stu- 
dents participating in community-based afterschool opportunities. The findings from these data show that students 
participating in community-based programs two or more times per week performed better academically in school 
than those children who did not. For more information, visit http://www.louisvilleky.gov/OYD/youth_print. 
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Boston, Massachusetts 

Mayor Thomas M. Menino’s Office, Boston After School & Beyond and the Boston Public Schools Department of 
Extended Learning Time, Afterschool and Services (DELTAS) collaborate to provide young people with high-quality 
afterschool programs. DELTAS funds and/or supports a growing network of school-community-family partner- 
ships, collectively known as the Triumph Collaborative. The management information system (MIS) for the Tri- 
umph Collaborative allows sites to track attendance, manage enrollment, access other student information (subject 
to confidentiality requirements) and track qualitative information about each child. Using MIS, DELTAS provides 
members with access to data, including academics (grades, test scores, school attendance, etc.), demographics, risk 
behavior statistics and program quality measurements. This information allows DELTAS to identify students for 
priority referrals to services across the city. 

To improve the MIS, a new feature was added in 2009 that enables the City of Boston and DELTAS to target students 
who should be involved in OST programs, rather than only focusing on the ones who show up. DELTAS will take the 
top 25 percent of students at highest risk and present them (through the MIS) to OST staff at each school. Doing so 
will allow DELTAS and OST programs to look more deeply into what challenges these students face and determine 
what actions the partners can take to decrease the students’ levels of risk and increase their academic success. 

Boston partners have also committed to establishing a citywide Results Framework for school-aged children and 
youth. The purpose of the framework is to strengthen the out-of-school time sector in Boston by ensuring that gov- 
ernment agencies and youth-serving organizations are collectively promoting, tracking and achieving the outcomes 
that matter the most for children and youth. The Results Framework and the data it yields could potentially be used 
to drive funding decisions and resource allocations. For more information, visit http://www.bostonbeyond.org. 


STRATEGY 2: CONDUCT MARKET RESEARCH TO ASSESS FAMILY, YOUTH AND 
COMMUNITY NEEDS. 


Market research is a process by which municipal leaders can collect, analyze and use information from out-of-school 
time stakeholders. A large part of market research involves listening to what youth and their families want, and 
actively using this information throughout OST planning. Collecting this information supports the development of 
effective programs and policies that best meet the needs of participants and the community. Market research can 
provide the following information: who needs programs or is most likely to use programs if they are offered; what 
kinds of programs are likely to be most successful in meeting youth and family needs; what barriers might prevent 
access to and participation in OST programs, such as affordability, lack of transportation or perceptions about qual- 
ity; and what programs already exist in the city. 

Why should municipal leaders conduct market research? Gathering this information can produce important benefits 
by helping cities: 

• Better allocate and use resources (e.g., time, money); 

• Inform and refine OST strategies based on factual data; 

• Raise awareness of the needs and preferences of children, youth and families regarding OST; 

• Build public will and secure “buy-in” from important stakeholders, such as schools, parents and community 
leaders; 

• Pursue new and creative programming ideas; 
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• Change OST programs when needed, or eliminate unwanted programs; 

• Influence public policy; and 

• Generate stronger evidence of the need for funding. 

Conducting market research may at first seem to be a daunting task, but this process provides municipal leaders with 
valuable information that can impact future planning and help ensure resources are used effectively. Market research 
can also help focus community partners and provide them with the data necessary to make more informed decisions 
about their cities’ unique OST needs. Cities can conduct market research at any point during citywide OST planning: 
prior to making decisions; during the planning process to determine a particular age group on which to focus or to 
identify potential funders; and during program operation to refine activities to assess impact. 2 

Cities have funded market research in a number of ways, many of which are quite affordable. While some cities may 
choose to contract with an evaluator using city funds, market research can also be conducted internally with existing 
staff capacity. Municipal leaders may also consider engaging a local research firm or university that might be inter- 
ested in donating their services and expertise to complete a market research project on the OST opportunities in the 
community. The decision on whether to contract with an external entity is often driven by the needs and resources 
of a particular community. There are a number of factors to consider when making this decision: Does the city have 
the funding to hire outside help? Does the city have the knowledge to collect, analyze and use the data collected 
effectively? Are there other resources in the city that would offer a cost-effective means of collecting the necessary 
information? 


Shoreline, Washington 

In late 2007, the City of Shoreline partnered with School’s Out Washington, a nonprofit, statewide afterschool inter- 
mediary organization, to complete a Supply and Demand study, which helped the community gain a better under- 
standing of what young people in Shoreline need during the afterschool hours. 

This study looked at three distinct communities: Shoreline, Tacoma and Lower Yakima Valley. As part of the study, 
these communities conducted surveys with OST providers and focus groups of parents and youth from across the 
state to measure what they look for in OST programs and their level of satisfaction with the current state of out-of- 
school time programming in Washington. The Bill & Melinda Gates Foundation, the Charles Stewart Mott Founda- 
tion, United Way of Pierce County, the state Office of Superintendent of Public Instruction and other community 
partners funded the study at a cost of more than $200,000. 

By listening to families, Shoreline officials learned that even though afterschool and youth development programs 
may have been open and had space for additional young people, families did not always know where to find them, 
needed help with transportation and were concerned about quality, hours of operation and cost. The study also found 
that families wanted extended program hours in the evenings, and that only 9 percent of the programs in Shoreline 
were open for more than four hours. The families also responded that they wanted more affordable or free programs. 
The majority of program fees in Shoreline exceed $200 per month. 

Based on the information collected for the Supply and Demand study, Shoreline leaders are taking steps to align com- 
munity resources to ensure that OST options meet the needs of all young people. The City of Shoreline responded 
to concerns expressed by families by completing a geographic information system (GIS) mapping project (this strat- 
egy is described more fully in the following section) to look more closely at the location of programs and schools 
within the city. The city is using this information to increase access and participation. For more information, visit 
http://www.schoolsoutwashington.org. 
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Nashville, Tennessee 

In 2009, Mayor Karl Dean announced the launch of the Nashville Afterschool Zone Alliance (NAZA). As the sole 
new initiative in the mayor’s proposed 2009 city budget — developed in the context of challenging fiscal constraints 
— the Metro Council appropriated $400,000 to launch the first afterschool zone, the Northeast Zone, in January 
2010. Metro Nashville Public Schools (MNPS) matched this funding with in-kind facilities, transportation and staff 
support. Community partners leveraged more than $100,000 through federal and grant funding. Chaired by Mayor 
Dean and vice chaired by the director of MNPS, Dr. Jesse Register, NAZA builds on the AfterZone model developed 
in Providence, R.I., to offer middle school students a range of OST opportunities in geographically defined “zones.” 

Prior to Mayor Dean’s announcement, leaders of NAZA identified and surveyed out-of-school time providers to 
better understand Nashville’s OST landscape. Recognizing the need for a strong youth voice during planning and 
implementation of the zone model, the city conducted surveys and focus groups with middle school students, along 
with parent and neighborhood groups, to gain their perspectives about what young people and their families desired 
from OST opportunities. The NAZA marketing and communications work group, all volunteers with a breadth of 
experience, developed the focus group questions. Members of this work group also donated their time to lead the 
focus groups and gather the data for this project. 

This process enabled Nashville youth and their families to play an active role in suggesting names for the zone model 
at their school and determining how to attract young people to participate in programs. Students reported that they 
would attend programs that offered sports and games, music recording, art, mixed media, fashion classes, mentor- 
ing, computer training and outdoor activities. During one focus group, the young people expressed concerns about 
students dropping out, the need for mentors and programs addressing the needs of homeless youth. NAZA made 
a concerted effort to engage young people who were not already involved in OST activities. These youth expressed 
interest in participating in a NAZA youth leadership committee to guide future actions of the zones. 

The focus groups gave students ownership of the programs in which they and their peers would participate. By 
listening to the youth voice, NAZA is now developing high-quality programming that is responsive to commu- 
nity needs, concerns and desires. The city plans to launch six or seven zones to cover the geographically diverse 
county, with the second zone launched in January 2011 with $600,000 appropriated by the Metro Council and 
$250,000 in additional funds from a local private foundation. NAZA’s Anchor Partners and Enhancement Part- 
ners have leveraged an additional $100,000 in funding and in-kind support each semester. The city is tying the 
success of the NAZA initiative to its efforts to increase high school graduation rates. For more information, visit 
http://www.nashvillez.org. 

STRATEGY 3: USE GEOGRAPHIC INFORMATION SYSTEM TECHNOLOGY TO IDEN- 
TIFY SERVICE GAPS AND IMPROVE ACCESS. 


Municipal leaders can use geographic information system (GIS) technology to get a clearer picture of a community’s 
OST landscape. This technology allows city officials to map and analyze data geographically, and can be a valuable 
tool for assessing the accessibility of local youth programs. Many city planning departments already use GIS, making 
this a cost-effective way to capture OST information. Additionally, local officials can use GIS to map key indicators, 
overlaying and comparing data from other city and community agencies with the location of OST programs. By 
sharing and combining GIS information, city, school and community partners are able to better understand how 
programs are distributed across the city and whether they have a relationship with other indicators (e.g., juvenile 
crime rates and school attendance by neighborhood). GIS equips municipal leaders with the information necessary 
to identify gaps, improve access and increase participation in OST programs. 
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Municipal leaders can promote greater use of GIS technology by convening key partners, directing city resources 
or departments to conduct mapping projects and advocating for the alignment of community resources. The 
information gathered from a GIS mapping project often helps to ensure that all young people have access to 
OST activities. Further, a GIS project can track other critical social services such as health and family supports 
to broaden a community-wide conversation about what services, beyond OST programming, youth in a specific 
neighborhood need. 

Boise , Idaho 

In May 2006, Mayor David H. Bieter formed the Mayor’s Council on Children and Youth (MCCY), which has col- 
laborated with the Boise Independent School District and other community partners to launch the Boise After3 
initiative. The council oversaw a survey of program providers, and used the city’s GIS technology to determine where 
programs are located in relation to school attendance areas, disinvested areas and juvenile crime locations. Based 
on these data, council recommendations led the city to reallocate funds and create a mobile recreation van that now 
brings parks and recreation OST programs, staff and equipment to Title One schools, parks in disinvested areas and 
housing for refugees. To increase access to OST programs, the city also entered into a partnership with the Boise 
School District to construct and operate community centers attached to three new elementary schools. For more 
information, visit http://after3.cityofboise.org. 

New Orleans, Louisiana 

Mapping New Orleans, a GIS data collection project, is a community engagement strategy to identify neighborhood 
resources and help young people and their families find available programs. The Greater New Orleans Afterschool 
Partnership, which is the intermediary organization leveraging resources to improve OST programs in New Orleans, 
sponsors Mapping New Orleans in partnership with Youthline America, a national organization whose mission is to 
“build a human connection in a digital world.” 

The GIS data collected for Mapping New Orleans are useful to neighborhoods, city agencies, other governmen- 
tal agencies and nonprofit organizations that seek to study patterns of resource distribution and develop new 
resources in New Orleans. Increased public knowledge of available resources offers all users a chance to visually 
demonstrate, in real time, where resources are lacking and needed. For example, in several neighborhoods, Map- 
ping New Orleans revealed that few or no recreational opportunities exist for young people. Municipal officials, 
community-based organizations, the press and residents can use this mapped data for planning, advocating and 
community building. 

Mapping New Orleans also allows youth to give meaningful feedback about the resources in their community 
in an informed and monitored way. Young people are selected, trained and paid to gather the data used to cre- 
ate an interactive Web-based communication system, neworleans.ilivehere.info. This online database populates 
a public website for the entire New Orleans community to use as a guide. The technology used for this project is 
open-sourced, live and kept up-to-date using software called “greenstat.” The data gathered for neworleans.ili- 
vehere.info provide an overall picture of the youth and family resources available in New Orleans and help local 
officials improve the distribution of resources in a way that will properly serve all neighborhoods. Youth and local 
leaders worked together to map the entire city of New Orleans by the end of 2010. For more information, visit 
http://gnoafterschool.org. 

Baltimore, Maryland 

With coordination provided by the Family League of Baltimore City (FLBC), a nonprofit intermediary, local leaders 
in Baltimore analyze youth risk indicator data, such as juvenile arrests, poverty and rates of chronic absence from 
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school, to identify specific Community Statistical Areas (CSAs) with the greatest needs. These CSAs are established 
as priorities for investment of out-of-school time programming funds. Applications from youth service providers 
that serve youth residing within those CSAs receive a competitive advantage over those that do not. 

By pooling resources from multiple public and private sources, FLBC has the capacity to provide funding and over- 
sight for 60 programs that serve more than 5,000 youth. In addition, the agency facilitates an OST Steering Commit- 
tee and supports Mayor Stephanie Rawlings-Blake’s Youth B’More initiative, which aims to ensure that youth have 
the resources (including nutrition, job opportunities, mentoring and OST programs) they need to maximize learn- 
ing and career outcomes. The city and school district are also working together to improve data sharing so they can 
better assess the impact of OST programs on student outcomes and develop targeted enrollment efforts. For more 
information, visit http://www.youthbmore.org. 

STRATEGY 4: DEVELOP, ADAPT AND IMPLEMENT A CITYWIDE PROGRAM 
QUALITY ASSESSMENT TOOL. 


“Getting kids in the door” by improving access to OST programs is certainly an important function of a city-level 
effort. If programs are not offering quality experiences for youth, however, they will not sustain the participation lev- 
els necessary for youth to reap the benefits of participation. Gathering detailed attendance data can be a useful initial 
step in gauging program quality. These data can reveal whether children and youth take part in various programs on 
a consistent or sporadic basis. 

Cities across the country are also using data to establish quality standards and to develop quality assessment 
tools that help program staff improve services for youth. Quality standards can convey clear expectations for 
programs; provide a way to assess what programs are doing well and where improvements are needed; offer a way 
for parents and young people to know what to expect from OST activities; and provide accountability mecha- 
nisms for funders. 

Several national organizations, statewide afterschool networks and cities have developed quality standards and 
assessment tools. A typical quality assessment tool identifies the key elements or standards that must be met for a 
program to be deemed effective, and provides OST partners with a method of measuring the degree to which their 
programs meet those standards. Assessment information can then be used to promote ongoing quality improvement. 
Communities can build on existing quality standards and assessment tools and adapt them to best fit their needs 
based on information gathered from providers, parents and youth. Drawing on existing program quality resources 
can save communities time and expense. 

Involving OST providers during the development of quality standards and assessment tools is critical to help pro- 
mote their “buy-in” and sense of ownership of the final product. Providers are often wary that additional standards 
and requirements will restrict their ability to implement programs. By approaching providers at the beginning of 
quality improvement efforts and requesting their input throughout the process, municipal leaders can gain greater 
support for quality initiatives as they move forward. 

Municipal officials can take a number of steps to support citywide program quality through better collection of data: 

• As described previously, the ability to track and monitor basic attendance and program data through an MIS 
will enable cities to identify participation patterns as a proxy for quality. 
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• Municipal leaders can work collaboratively with stakeholders, including families and youth (e.g., through 
focus groups, surveys or interviews), to identify key components of program quality for their community, and 
articulate a set of common standards for all OST programs in the system. 

• Municipal leaders and partners can develop or adapt a program quality assessment tool that best measures 
programs’ adherence to quality standards. The tool could be a hybrid of several other tools, and the box below 
provides some guidance on how to select and/or develop a citywide quality assessment tool. 

• Finally, a city can use quality assessment information to target program quality improvement and professional 
development. 

Successful implementation of these steps, however, requires that programs and municipal leaders have the capacity 
to use the tools effectively to both get good information and understand and use the information that such tools can 
generate. Some additional training and capacity building, supported by the city, may be needed so that municipal and 
program leaders know how to use information on program quality to improve local OST programs. 


Guiding Questions for Selecting or Developing a Citywide Quality Assessment Tool 

When stakeholders meet to discuss the adoption and implementation of a citywide program quality assessment 
tool, they should ask themselves the following questions: 

• Why do we want to develop a quality assessment tool? 

• What are the primary goals of our city wide OST initiative? 

• What aspects of quality are we most concerned about? 

• If we are considering adopting a pre-existing tool, for what purpose was that tool developed? 

• What tool format will work best for the programs in our city? 

• Who will use the program quality assessment tool? 

• What are the costs of developing and implementing the tool? 


Tampa, Florida 

Mayor Pam Iorio has worked with Hillsborough County Schools Superintendent MaryEllen Elia and the 
Children’s Board of Hillsborough County to improve afterschool opportunities for students. As part of 
this work, more than 160 community agencies came together in 2006-07 to form the Partnership for Out- 
of-School Time (POST) collaborative. POST developed and continues to focus on the following vision: 
“Every child/youth will have high-quality, out-of-school time opportunities that are accessible, inclusive 
and accountable.” 

One of POST’s goals is to ensure that OST programs and services in Hillsborough County use the highest quality 
practices and standards. A subcommittee on quality has researched and pilot tested several models for measuring 
quality at OST sites in the city and county. In addition, POST assisted the Florida Afferschool Network (FAN) in 
the development of FAN’s quality standards and self-assessment tool. These standards for Quality Afterschool Pro- 
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grams define the principles and practices that lead to the delivery of quality programming for children in elementary 
school. The FAN standards are organized in seven categories: 

• Administration and organization 

• Program management and staff 

• Communication and interaction 

• Program structure and activities 

• Health, safety and nutrition 

• Program environment 

• Family and community involvement 

Several POST members and afterschool sites participated in developing and testing the FAN standards and self- 
assessment tool. Implementation of the standards is voluntary, but FAN hopes that program providers see the stan- 
dards as valuable and implement some or all of the elements. FAN also envisions parents and guardians using infor- 
mation from the self-assessments as a guide when selecting an afterschool program for their children. For more 
information, visit http://www.childrensboard.org. 

Providence, Rhode Island 

In 2004, the Providence After School Alliance (PASA) convened afterschool providers, youth, parents and funders 
as an initial step in their quality improvement strategy. Using quality afterschool standards developed by other cit- 
ies and national organizations as a foundation, PASA and these afterschool providers identified the most important 
aspects of quality and what they should be working toward to provide the best programs for young people in Provi- 
dence. This work led to the development of the Rhode Island quality standards for afterschool programs, which con- 
tain five major categories: health, safety and environment; relationships; programming and activities; staffing and 
professional development; and administration. 

In order to make the quality standards more useful and practical, PASA partnered with the High/Scope Educational 
Research Foundation to develop the Rhode Island Program Quality Assessment Tool (RIPQA). The RIPQA com- 
bines High/Scope’s nationally validated Youth Program Quality Assessment with a supplementary component that 
addresses the unique needs and priorities of Rhode Island afterschool providers. In 2007, a variety of afterschool 
providers pilot tested and vetted the RIPQA, which has since been adopted statewide through a partnership among 
PASA, Rhode Island After School Plus Alliance (RIASPA) and the 21st Century Community Learning Center (21st 
CCLC) initiative of the Rhode Island Department of Education. All 21st CCLCs across Rhode Island and more than 
60 of PASA’s program partners in Providence are currently using the tool. Each site is assigned a “quality advisor” who 
observes programs, facilitates quality improvement planning and provides one-on-one technical assistance to each 
site as needed based on the recommended action steps. PASA continues to develop its quality improvement system 
in partnership with the David P. Weikart Center for Youth Program Quality, a joint venture between the High/Scope 
Education Foundation and the Forum for Youth Investment. For more information, visit http://www.mypasa.org. 

STRATEGY 5: CONDUCT AND SUPPORT PROGRAM AND CITY-LEVEL EVALUATION 
EFFORTS, INCLUDING CROSS-SYSTEM DATA COLLECTION. 

Credible evaluation information is an essential component of a citywide strategy for strengthening and sustaining 
OST programs and systems. Evaluations at the program and city levels assess the impact of and provide feedback 
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on OST programs. Given the new economic climate in which cities are operating, municipal leaders are increas- 
ingly moving toward data-driven, results-based decision making, and evaluations can provide information to help 
guide decisions. Municipal leaders can require and support the collection and use of evaluation data at two levels: 
(1) at the individual program level and (2) at the city level through a collective citywide evaluation effort. 

At the program level, evaluations can feed into city-level accountability systems and support policy/advocacy 
work and sustainability. Specifically, OST programs should be encouraged to conduct evaluations for at least three 
reasons: 


• To make management decisions - Evaluation data help OST system leaders make internal management deci- 
sions about what is and is not working, where improvement is needed and how to allocate scarce resources. 

• To demonstrate accountability - Evaluation data demonstrate to current funders that their investment is 
yielding the intended results. Evaluation results can also be used in marketing materials such as brochures or 
published reports that help to promote a program to potential participants, the media, potential funders and 
other community members. 

• To build a case for sustainability - Evaluation results can show the impact a program has had on an indi- 
vidual, family, school or neighborhood, and thereby strengthen the case for future funding. 

Municipal leaders can lead evaluation efforts by expecting programs to operate with a sound theory of change that 
describes how they will achieve their intended results. A theory of change is a progression of thinking and planning 
that guides a team as it sets program goals and designs program elements that can result in positive participant out- 
comes. 3 Developing a theory of change is also useful for evaluation planning, continuous learning and improvement 
and effective communication among diverse program partners. A key feature of developing a theory of change is 
to engage local afterschool partners, including program staff, program participants, their families and other com- 
munity members and organizations, in the development process. Getting “buy-in” from all parties ensures that the 
program will be grounded in the ideas, beliefs and principles of the community. 

While municipal leaders most likely will not conduct or manage local program evaluations, they can establish city- 
wide expectations for how information should be collected. For example, municipal leaders can set the stage for pro- 
gram evaluation by instilling an “evaluation mindset” among the programs they support and other program provid- 
ers throughout the city. Municipal leaders can foster this attitude in a number of ways, including: clearly stating that 
they expect all programs to collect basic data on participants and programs; using the data collected for continuous 
learning and accountability purposes; and ensuring that sufficient resources for evaluations are built into program 
budgets by requiring a set-aside to be used for evaluation purposes. 

At the city level, some local officials conduct citywide evaluations to support data-driven decision making about fund- 
ing priorities, craft new requests for proposals and build public will and support for OST. Citywide data collected on 
student attendance, levels of participation, engagement in particular activities and social- emotional indicators can 
improve social marketing and branding, modify program recruitment strategies and inform OST curriculum and 
planning. However, gaining access to data is often challenging, and conducting a city-level evaluation requires city 
leadership and sufficient resources to support cross-system data-sharing. Facilitating collaboration among school 
and community partners to share participation and outcomes data, including academic and attendance records, is a 
key role that municipal leaders can play. 

Part of the challenge municipal leaders may face when collecting and using information involves understanding 
the potential for learning from existing information and deciding what new information must be generated. There- 
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fore, prior to launching efforts to generate new information, municipal leaders can conduct a thorough inventory of 
what information is already available. This information is often easily accessible and might include: U.S. Census data 
on children and families; research studies conducted by local businesses or organizations; results of individual pro- 
gram evaluations; literature reviews and research syntheses; local media coverage of OST; and national data. Using 
existing data is a cost-effective way for community leaders to collect an initial snapshot of OST activities in their city. 

After an examination of existing data, municipal leaders can enlist community partners to decide what new data 
should be collected to inform decision making and the most efficient process that can be used to gather these data. 
Municipal leaders can use a variety of tools to collect information, including: surveys of providers, parents and youth; 
focus groups; and telephone or face-to-face interviews. Because circumstances and needs vary by city, it is important 
to consider which type of research will help local decision makers make the most informed choices about a com- 
munity’s OST options. 

Conducting evaluations can be costly and a lack of resources may deter cities from considering an evaluation project. 
However, there are cost effective options that municipal leaders can use to collect evaluation data, such as engaging 
institutions of higher education. Colleges and universities can play an important role in conducting evaluations and 
analyzing data at the program and city levels. Municipal leaders can work to engage local universities in conducting 
evaluations and help demonstrate to other stakeholders the value of postsecondary institutions’ involvement. Often, 
colleges and universities will donate their expertise to conduct an evaluation of OST initiatives in a community. Not 
only does the involvement of colleges or universities provide a cost savings for the city, but their participation also lends 
credibility and takes advantage of evaluation expertise that program staff often do not have. There may also be other 
local entities that municipal leaders can engage to provide low-cost or no-cost evaluation services. City officials may 
want to look to the business community to see if a local research firm will donate its services as an outside evaluator. 

Akron, Ohio 

In 1998, Mayor Donald L. Plusquellic initiated the development of afterschool programs based on the federal 21st 
Century Community Learning Centers model. To support these programs, Akron Public Schools (APS) created an 
afterschool staff position that was supplemented with seed money from the City of Akron. Currently, these after- 
school programs are operating in all 27 elementary schools and three middle schools, with plans to expand. Since the 
programs’ inception, an outside evaluator has conducted program evaluations that consistently show positive results. 
The data collected in these evaluations help programs demonstrate their accountability, giving them credibility in 
the community. 

In 2009, using data from several standardized tests that measured math and reading achievement for students in 
grades K-5, the Kent State University Bureau of Research Training and Services conducted an evaluation of the 
APS elementary afterschool programs. The study, supported by city funds, found that academically at-risk children 
who were referred to APS afterschool programs and attended regularly scored the same or higher on standardized 
reading and math tests than students who only attended occasionally as well as their less at-risk peers who did not 
attend at all. This evaluation underscored the importance of continual investment of the resources necessary to 
provide these programs consistently during the school year. In nearly every measure of academic performance, 
students who attended 30 or more times per year and therefore received higher amounts of intervention had higher 
test scores and showed greater improvement than students who attended fewer than 30 times during the 2009-10 
school year. 

Akron’s evaluation process includes: 

• A comprehensive needs assessment process that guides the overall afterschool program design; 
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• A needs assessment that uses multiple strategies to collect data (e.g., surveys, interviews, focus groups, etc.); 

• Multiple types of data that are collected to inform out-of-school activities and monitor progress on grades, 
homework, completion rates and disruptive behavior. For more information, visit http://www.akronschools. 
com/programs. 

Fort Worth, Texas 

Fort Worth After School (FWAS), an afterschool initiative funded jointly with city and school district contributions, 
works with Texas A&M University to conduct an annual, comprehensive program evaluation. FWAS has developed 
evaluation tools to measure the effectiveness of OST programs, encourage continuous improvement and improve admin- 
istrative processes. Data are collected from participants and their parents, service providers and staff, school principals 
and FWAS leadership. Based on the evaluation reports, FWAS staff hold an annual conference to train site coordinators 
and program staff on the desired outcomes, required standards and new ideas to implement high-quality programming. 

By conducting similar evaluations each year, Fort Worth has seen continuous improvement in program processes, 
but more importantly, FWAS partners have a multi-year data set to help determine whether the programming is 
making a difference over time in the lives of the children they are reaching. The availability of strong data helps dem- 
onstrate to the community that a public investment has a substantial impact on the children and youth participating 
in OST activities. For more information, visit http://www.fwisd.org/fwas. 

STRATEGY 6: DEVELOP COMMON, SYSTEM WIDE OUTCOME MEASURES AND INDI- 
CATORS. 

Adopting and publicly reporting against a common set of outcomes serves a number of purposes, including sup- 
porting citywide evaluation efforts as described above. In addition, developing common measures and outcomes has 
other benefits for cities: it can help programs benchmark against city-level outcomes; hold providers accountable for 
program quality; foster collaboration; and build a common advocacy base for OST sustainability. Developing com- 
mon measures across a citywide OST system entails convening all partner organizations at the city level to discuss 
the common goals toward which all programs in the city should be working. Municipal leaders can play an important 
role in convening multiple stakeholders, including school leaders, business leaders and community-based organiza- 
tions. Given cities’ increasing focus on expanded learning time, it is important that municipal leaders include school 
district leaders in the discussion when they convene partners to talk about common OST outcome measures. 

Many cities that are advanced in their data collection efforts are beginning to look at how to improve community- wide 
outcomes for young people through participation in OST programs. Municipal leaders can initiate a discussion about 
how OST outcomes relate to other city priorities. For example, OST programs can help to lower juvenile crime, decrease 
the dropout rate, prevent childhood obesity, prepare youth for postsecondary education and the workforce and pro- 
mote economic development. Using information to demonstrate the impact that youth participation in OST programs 
has on community outcomes can build public will to support OST initiatives. In addition, using data to benchmark 
OST programs against other city initiatives helps municipal leaders justify continual investment in these programs, 
learn more about what works and understand how the city’s OST initiatives impact other goals and priorities. 

While common, system-wide outcome measures are in their nascent stages of development, some cities have begun 
to articulate a set of common measures for OST (see sidebar). Results of their efforts suggest that at the youth level, 
outcomes might relate to school attendance and skill mastery; at the program level, outcomes might relate to program 
quality features such as staff ratios and activity characteristics; and at the systems level, outcomes might relate to use 
of program quality assessment tools and improved access to participation. 
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Collaborative for Building After School Systems (CBASS) Measures for Assessing Afterschool Services, 

Programs and Systems 

Systems Level 

• Use of participant tracking system 

• Partner agreements and policies in place 

• Adoption and use of quality standards 

• Provision of technical assistance and training 

• Sustainable financial support 

• Unified governance structure 

• Growth in number of program slots 

• Engagement of leaders across sectors 

Program Level 

• Low staff-youth ratio 

• High educational levels of director and staff 

• Outreach to parents 

• Youth input into program design 

• Positive relationships with adults and peers 

• Opportunities for activity choice and leadership 

• Explicit activity sequencing 

• Active, hands-on learning opportunities 

• Breadth of content and activities 

Youth Level Outcomes 

• High sustained daily program attendance 

• High year-to-year retention 

• High daily school attendance 

• On-time grade promotion, leading to high school graduation 

• Mastery of academic and non-academic skills 

Adapted from The Forum for Youth Investment (2008). Out-of-School-Time Policy Commentary #13: Speaking in One Voice: Toward 
Common Measures for OST Programs & Systems. Washington, DC: The Forum for Youth Investment. Retrieved from http://www. 
forumforyouthinvestment.org/files/OSTPC13.pdf. 
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Grand Rapids, Michigan 

With the support of Mayor George Heartwell, Our Community’s Children — formerly known as the Mayor’s Office 
of Children, Youth and Families, and now a public/private partnership of the City of Grand Rapids, Grand Rapids 
Public Schools and various community organizations — has made impressive strides in expanding access to after- 
school programs and ensuring they meet high quality standards. 

To measure the impact of afterschool programs on other community outcomes and city priorities in Grand Rapids, 
Our Community’s Children partnered with the Grand Rapids Police Department and the Community Research 
Institute at Grand Valley State University to share afterschool data for the city’s March 2009 Juvenile Offense Index 
Report. The report found that youth crimes occur overwhelmingly in the hours immediately after school on week- 
days, as well as during the evening on both school and non-school days. Additionally, a pilot study found that after- 
school program participants were unlikely to be involved in juvenile offenses. The city and other stakeholders are 
using these data to highlight the need for adequate OST programming as a method of increasing public safety. For 
more information, visit http://www.grand-rapids.mi.us/3501. 

Denver, Colorado 

A seven-year longitudinal study of Denver’s afterschool programs has consistently demonstrated a strong correla- 
tion between program participation and improved academic outcomes. This study found that participants of out- 
of-school time programs have lower dropout rates, increased school attendance and higher standardized test scores. 
Notably, this information has shown that the dropout rate for afterschool participants is five percent lower than 
for nonparticipants on an annual basis. The study also found that multiple years of participation is associated with 
increased school engagement and academic achievement. 

The 2009-2010 evaluation of Denver’s OST programs included a survey of students who regularly participated 
(attending 30 days or more) in afterschool programs during the 2009-10 school year, and found an improvement in 
pro-social behaviors. Ninety-two percent of the students reported that afterschool participation helped them better 
interact with adults, 91 percent reported that afterschool activities helped them feel safe, and 90 percent reported that 
these programs improved their self-confidence. 

Under the leadership of former Mayor John Flickenlooper and with the support of the City Council, the City 
of Denver has built a strong partnership with Denver Public Schools (DPS), Mile High United Way, the Denver 
Public Schools Foundation and other community-based organizations to leverage resources to advance quality 
afterschool programs and continue to improve students’ academic achievement. This partnership, the Lights 
On After School Initiative (LOAS), seeks to make efficient use of resources, promote high quality and better 
train afterschool staff to provide strength-based afterschool programs at every DPS elementary and middle 
school. 

LOAS has also collected and used data to design and implement free staff development trainings for all Denver 
afterschool providers that focus on positive youth development. All LOAS-funded programs must meet mini- 
mum quality standards tied to positive youth development principles, with developmentally age-appropriate 
activities that promote “cultural competence” among youth, are aligned with school curricula and have a low 
staff-to-student ratio. Programs funded by LOAS must also report the number of students served, number of 
program hours and student academic and developmental outcomes. By consistently showing afterschool pro- 
grams’ positive impact on young people’s academic and social growth, LOAS partners have helped to attract 
funding from local, state, federal and private sources. For more information, visit http://www.denvergov.org/ 
educationandchildren. 
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CONCLUSION 

The strategies described above highlight specific actions that municipal leaders can take to ensure that information is 
collected and used well. This work can be challenging, but with municipal leaders as advocates, cities small and large 
have the opportunity to use strategies presented here to listen to community voices, identify gaps in services, collect 
participant data and develop community outcome measures. 

As city budgets become tighter, systems and programs are understandably under pressure to be more accountable, 
and accurate information is essential to capture and publicize the benefits of high-quality OST opportunities. Many 
of the city examples provided in this guide offer details on the array of resources that can be used, often in combi- 
nation, to support data collection efforts. The cost of this work can vary greatly, from inexpensive to quite costly, 
depending on the goals pursued and strategies employed. City revenues, donated staff time, outside contractors, 
private funders, in-kind donations and other community resources have all been utilized by cities across the nation 
in creative efforts to support data collection projects. 

Making investments in the collection and use of data allows municipal leaders to make better decisions, hold pro- 
grams accountable and enhance their effectiveness. When municipal leaders support the collection and use of data, 
they help the OST opportunities in their communities reach a higher level of quality and develop the 21st century, 
social, cultural and personal skills necessary for young people’s success. 
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Bodilly, Susan and Megan K. Beckett (2005). Making Out-of-School Time Matter: Evidence for an Action Agenda. 
Santa Monica, CA: RAND Corporation. http://www.rand.org/pubs/monographs/2005/RAND_MG242.pdf 
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